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Student publishing of value-added contributions to a digital library 

Abstract 

Students can make authentic contributions to the WWW and learn critical evalution skills by publishing reviews of 
WWW resources. This paper will describe a project where high school students wrote and published reviews of 
resources they used as part of an inquiry-based science curriculum. Students were supported in writing reviews by 
specialized software , and by in-class modeling of the review process. We were generally pleased to find that high 
school students could critically evaluate sources and produce written analyses that synthesized their evaluations. 
However , student products were inconsistent , and we were not successful in helping students recognize separate 
types of critical evaluation (e.g. evaluation of source versus analysis of evidence). Future developments in both 
software and instructional support of WWW reviews are discussed. Reviews published in this project are available 
online from [http://chs-web.neb.net/pollution]. 



Introduction 

K-12 students can 'give back to the library' by writing and publishing reviews of resources that they have 
found helpful in their research. There may be a wide range of interesting benefits to having students 
produce reviews and other Value-added' contributions to a library. This paper will begin to explore 
those benefits by analyzing one set of reviews written by high school students in the context of a project- 
based science class (Blumenfeld, et al., 1991). Students located resources relevant to their topic, mostly on 
the WWW but also a few offline resources, and published 'reviews' of these resources on the WWW for 
the benefit or other researchers. We will describe both the methods we made to scaffold student-written 
reviews, and perform content analyses of the reviews that were published to determine what parts of this 
activity were successful and what challenges surfaced. 

Benefits of WWW reviews 

Student-published reviews have several possible benefits for both the students and the digital library. 
First, students can learn a range of critical evaluation skills by writing reviews of resources. Second, 
student-published reviews can add value to a digital library (e.g. the WWW), enabling other users to 
benefit from the searching and evaluating done by students. Third, value-added contributions represent 
an authentic publishing task which may help engage students, increase their ownership of the library, 
and help their understanding of how scientists evaluate and use information. 

Critical evaluation of WWW resources 
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The WWW is a good place for students to learn how to critically evaluate source materials. This quality 
of the WWW is bom of difficulty. As the other papers in this symposium will attest (Wallace, 1997), the 
WWW is a trying medium in which to conduct research, because of its vastness, it s lack of central 
organizing features, the uneven and unpredictable nature of its content, and because it is ever-changing. 
We are not the first researchers to recognize that this difficulty also presents opportunity for teaching 
critical analysis and other ’information-age* skills. 

Although we hope that the WWW will not always be as confusing a place as it currently is for K-12 
students, the need for some critical evaluation skills is probably an enduring feature of the WWW, and of 
digital libraries in general. Distributed and decentralized digital libraries, by their nature, will always 
offer content that varies widely in source, reliability, technical level, media type, organziational structure 
and purpose. In order to move within these complex information spaces, students must have well- 
developed 'meta' level skills in evaluating resources and judging what is and is not relevant to their topic 
area. Therefore, it will be increasingly worthwhile to help K-12 students learn to critically evaluate 
materials. 

What are the components of critical evaluation? Too often critical thinking* is thought to be a single type 
of analysis, when it actually involves a range of distinct types of analyses. In the project described in this 
paper, we asked students to evaluate sources according to 1) types of scientific claims made, 2) scientific 
credibility of those claims, 3) source of the published resource, 4) appearance of the resource and5) the 
organizational structure of the resource. We also asked students to compose a synthesis of these separate 
analyses that would serve as a sort of abstract for other users. (More detailed description of the 
publishing task will follow.) 

Benefits of value-added user contributions to the library 

Student-published reviews are an example from a class of Value-added* contributions by users that may 
serve a crucial role in digital libraries of the future. In a traditional library evaluation of content is done 
first by publishers and later by librarians, and users have essentially no role. The same is true for the 
various ways that content is organized and made accessible within the library. In a dynamic and 
distributed digital library, these traditional gatekeeping functions are harder to fulfill, and users may 
have an important role. No publishers, librarian, or government controls the content of the Internet- 
even if they had the power, the content is too vast and changes too rapidly. Likewise, there are no 
centralized controls over how the Internet's content is organized, indexed, and accessed. In this 
environment, if a user takes the trouble to find and evaluate resources that are of use to them, it makes 
good sense that this Value added’ work persist in the library in some way, so that other users may benefit 
from it. 
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Various research groups, including the University of Michigan Digital Library, are developing ways that 
users can give back to the library by having their search results, evalutions, or written comments persist 
in some way that other users can access them. The WWW reviews that are the subject of this research are 
but one example of class of value-added contributions that may become increasingly important to digital 
collections. (Birmingham, et al., 1996; Ibsen, 1995; Terveen, Hill, Amento, McDonald,& Creter, 1997). 

Authentic publishing task 

Composing and publishing resource reviews can be an authentic writing task where students can practice 
skills of synthesizing content and writing for an outside audience. It is sometimes difficult to convince 
students to invest energy in writing projects that do not have a real communicative purpose. It may be 
easier to motivate students to critically analyze resources, and synthesize these analyses, if WWW 
reviews have an authentic role to play in digital collections. Also, the task of writing for a real audience 
may cognitively benefit students, by making them more conscious of the information needs of scientific 
readers. (In fact, it was for this last purpose that WWW reviews were originally developed, as a means of 
instructional support for student publishing of other materials. See Bos, Krajcik, & Soloway, 1997). 

Description of the project and instructional support 



Setting 

The setting for this study is the third-year cohort of an integrated science curriculum at Community High 
School in Ann Arbor, Michigan. This high school accepts students through a combination of lottery and first- 
come-first-serve signups each school year. The science curriculum in this school, called Foundations of Science 
(FOS), follows the principles of Project-Based Science (Blumenfeld, et al, 1991), and has a heavy emphasis on the 
use of educational technology. The science curriculum integrates the three traditional content areas of Earth 
science, biology, and chemistry into one three-year sequence focused on investigative science. 

Three sections of eleventh grade students published WWW reviews as part of their assigned work for the 
beginning of a project on air pollution. The final product of this unit was a research report on students' local air 
quality monitoring efforts, which were also published on the WWW (Krajcik, Bos, and Soloway, 1997). 

Students published WWW reviews early in the project, as they were conducting the background research for 
their own reports. The WWW reviews were enhanced because students had an authentic purpose for finding 
and evaluating useful WWW sites. The stated audience for the published reviews was other high school 
students who might be conducting research on a similar topic. 

Students were encourages to work in groups, but were required to publish enough reviews to average one per 
person for the group (i.e. a group of three would be required to publish three reviews). 
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Software support for WWW reviews 



Students published reviews by filling in an html form, [http://chs-web.neb.net 

/pollution/ reviews /con tribute.html] which solicited comments in seven categories. These categories 
were: scientific claim, evidence for claim, citation, source, organization of page, appearance of page, and 
overall description of resource. These categories will be discussed in detail in the analysis section. After 
students submitted a review, a cgi script parsed the student reviews and published sections of the review 
in two places. First, the script added summary information from the students' review to a table of the 
class' reviews (figure 1). Second, the script published the full version of each student review on a 
separate page, linked to the class table (figure 2). 
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figure 1 

Summary table of reviews published by one 11th 
grade class. Table entries show the description 
field along with 'hot' links to the full review and to 
the WWW resource itself. 



figure 2 

Full review of a WWW resource published by one 
group of students in the air pollution project. 
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This review-publishing was supported on the high school’s own Macintosh server, although it could as 
easily have been supported on another Macintosh server at a remote location. We used Maxima's 
Netforms software which allows non-programmers to parse the output of html forms and write the 
contents to WWW pages or other text files. The contents of the scripts that we designed and the basic file 
structure of the review pages are available on-line from the air pollution project site [http://chs- 
web.neb.net/ pollution/ about/ about.html]. Supporting WWW reviews in this way requires purchasing 
of the NetForms software ($136, available for trial download from http://www.maxum.com). 

We are currently designing a more sophisticated software tool, ShaReview, which writes reviews to a 
database, from which student-published reviews can be searched, edited, and linked to student 'home' 
pages. 

Instructional support for WWW reviews 

We provided two kinds of instructional support for students writing WWW reviews. The first level of 
instructional support was in the design of the form itself, which directed students attention to different 
aspects of the WWW source, differentiated separate kinds of evaluations, and directed them to synthesize 
comments. The second level of instructional support was a modeling exercise, where students wrote a 
short practice review of a real WWW site and then compared their reviews to a model review of that site. 

Design of form: Rather than present students with the daunting task of composing a free-form reviews, 
we instead devised a form divided into categories. Under each of the category titles were'prompting' 
questions. We were hesitant about using questions, because we wanted to avoid presenting the task as an 
online 'worksheet' which would encourage students to use their well-practiced skills of dispatching 
series' of schoolish questions with minimal answers. We tried to present the reviews as a true 
composition task requiring some thought as to presentation of information, by stressing the need to write 
in complete sentences and to anticipate the needs of WWW readers. 

The form was designed to support a set of high-level skills. By separating critical evaluation into six 
separate categories we encouraged differentiation — the conceptual separation of closely related categories. 
The form also prompted evaluation, which is a 'meta' level skill that is not often required of high school 
students. Writing a good reviewalso required summarization , intelligently condensing an often- 
unwieldy amount of information into a few sentences. Finally, the form prompted a thoughtful synthesis 
of information from the first six fields into a single description field. 

Practice reviews and modeling: Students were introduced to writing WWW reviews with a practice 
review activity. Students were given about 20 minutes in class to complete this sample review. After 
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students had some time to write their own reviews, the teacher displayed a sample review written by the 
researcher. 

The practice site for this sample review 1 was chosen because it illustrates both the good and the bad of 
WWW resources. The site, "Breath Taking" provided extensive, relevant, and easily referenced data, but 
did not provide adequate explanation of the source or credibility of this information. Students could 
access particulate pollution figures from many US cities, along with estimated deaths from particulate 
pollution. However, the source of these figures was an unspecified "study by the American Cancer 
Society and Harvard University", with no accounting for data collection methods, and more seriously, no 
explanation of how the estimated death counts were obtained. Also, the site was not arranged into a 
recognized scientific format, so that students had to hunt for what information there was related to source 
and methods. Modeling a review of a resource with these difficulties, which are very typical of the 
WWW in its current state, would make students aware of the importance of evidence and citation, as well 
as serve to illustrate for students the importance of citation, evidence, and predictable organization 
structure in scientific reporting. 

Analysis and discussion of student-published reviews 

Students published 51 reviews in all, over a period of approximately one month. Average length of 
reviews published was 9.7 lines, approximately 100-150 words. 

This project served as a pilot study for the WWW-publishing concept. All involved parties- teachers, 
students, researchers, and software developers, were unsure of exactly what the expectations for a high- 
quality student-published review should be, what sorts of resources students could find, and how well 
the software would work. The published products were, not surprisingly, inconsistent. Despite this, we 
were pleased to find that students could write worthwhile reviews, could analyze both content and 
evidence (to some degree), and could synthesized different categories of comments into short 
descriptions that could be genuinely helpful to other WWW users. 

Summary versus review 

One of the first questions we wanted to ask was, were the reviews' we asked students to write really 
different from any other school writing assignments? To study this we compared the published reviews 
from one class with a set of ten article summaries written by students in the same class. These summaries 
were assigned by the teachers at approximately the same time as the reviews, and were of articles similar 
in length and focus to the types of articles students were reviewing online. 

1 "Breath-taking: premature mortality due to particulate air pollution in 239 American cities" [http:/ / 
www.nrdc.org/ nrdcpro/bt / tableGu.html] 
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We were particularly interested in how much science content information was in each type of writing 
assignment, (we would expect more in the 'summaries’), and how many 'meta' level statements were in 
each (we would expect more in the reviews.) By meta’ level statements we mean statements that in some 
way go outside the scope of the immediate article, and put the article into a larger context by evaluating it 
or comparing it with knowledge from other sources. 

We were correct in our expectations for the relative occurence of content statements and 'meta' statments. 
On the average, the summaries were longer and contained more content-specific information. The 
'reviews' were shorter, contained less content information, but contained more 'meta' level statements 
about the articles. A few striking differences: in the ten article summaries, the source of the article itself 
was never mentioned. By contrast, 9 of 20 of the class’ reviews at least mentioned the source (and in the 
future we hope this will be much higher— see next section.) Many of the summaries did contain some 
'meta' level statements, such as student commentary about issues in the articles. However, none of the 
ten summaries included any commentary on the article itself, (e.g. "this article give good basic 
information about carbon dioxide") while almost all (19/20) of the reviews did include some commentary 
at this level. 

The WWW reviews, then, demanded less detailed content engagement by the student, but encouraged 
more high-level evaluation. What is the value of this higher 'meta' level of engagement? We believe that 
putting the content of a WWW page into a larger context has several possible benefits. First, it may help 
students form a larger and more accurate picture of the scientific world, which goes beyond isolated facts 
from nameless scientists, and envelops the researchers, government workers, and private enterprises that 
use scientific information for disparate purposes. Second, a larger picture is necessary for students to 
move in the complex information landscapes of the WWW. For example, students who have only ever 
been taught to process textbook knowledge are easily lost when presented with a list of hundreds of 
search engine 'hits' which must be rapidly classified and evaluated. Third, taking a 'meta’ view of 
scientific information can help students learn and remember new information by connecting it with their 
personal opinions and their knowledge from other domains. 

Content analysis of reviews 

We will now separately analyze each category of the review form. Along with analysis of each section, 
we discuss issues which arose in this project that may be broadly applicable to student-published 
reviews. 

'Claims’ field 
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Students summarized the scientific content of a WWW resources using the ’Claims’ category. On the 
average, students wrote about two lines of text summarizing claims. To determine how well students 
were identifying the claims of the WWW pages we were reviewing, we conducted a content analysis of 
one class’ published reviews (n = 14). We accessed the original WWW pages they had reviewed and rated 
what the students had written about the claims in three categories: accuracy, centrality, and 
comprehensiveness, with results presented in figure 3. 



Accuracy 


Centrality 


Comprehensiveness 


6 completely accurate 


4 identifies central claims 


1 comprehensive description of 
important claims 


7 accurate 


6 partially identifies central claims 


11 partial description of important 
claims 


1 not accurate 


4 poor identification of central 
claims 


2 poor description and coverage of 
claims 



figure 3 Accuracy, centrality, and comprehensiveness of claims (n=14) 



Most of what students wrote as claims were accurate, with only one student reporting a claim that was 
actually inaccurate. Given the variablity and high technical level of some of the sites reviewed, we were 
encouraged by this result. 

Students had more difficulty identifying what were the most central claims of a WWW source. This 
difficulty is not unexpected, given that students were at the beginning of a project, working in a content 
area that was mostly unfamiliar to them. When content is unfamiliar, it is more difficult for students to 
identify which claims are new, and which are common-knowledge background information. 

Nevertheless, students sometimes missed central claims that we felt should have been discemable, such 
as one instance where the central claim appeared in the resource title. We expect that with more practice, 
students should be able to get better at identifying the central claim of a resource, even when working in 
unfamiliar content areas. 

We also felt that students could do a better job of covering the breadth of claims at the sites they 
reviewed. Often, students would give only high-level overviews of the claims at a WWW site, (e.g. ’’This 
site makes claims about the health effects Carbon Monoxide”) when they should have gone into more 
detail (e.g. "This site describes how carbon monoxide robs the blood of oxygen, tells what the symptoms 
are, and tells how carbon monoxide poisoning can be prevented.”) Again, this is partially due to students’ 
unfamiliarity with the content. If students does not know that carbon monoxide has health effects, then 
that claim alone is a revelation and the rest may seem mere detail. By contrast a teacher or research 
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reading the summary is well aware that carbon monoxide is poisonous, may consider that fact too 
obvious to be considered a claim, and will want the student to delve into more detail. Students in this 
project were almost surely capable of describing claims in more detail, if pushed to do so. Future WWW- 
publishing projects will give more specific guidelines, perhaps in the form of length requirements, of how 
much claim coverage is expected. 

'Evidence' field 

One of the key skills necessary for using resources in an uncontrolled environment such as the WWW is 
the ability to evaluate credibility of sources. This is not a skill that most high school students are required 
to practice, because most information they are exposed to comes from carefully screened sources such as 
textbooks or school library collections. WWW reviews can give students an opportunity to develop 
savvy and skills in evaluating credibility. 

Analyzing the reviews published in this pilot project, we saw that students were able to do some 
evaluation of credibility. However, students seemed unable to draw distinctions between different 
methods of establishing credibility. We wanted students to evaluate source credibility in three related 
but distinct categories: evidence, citation, and source. Our intention was that in the ’Evidence’ category 
students would report whatever they could find about the scientific data-collection behind the 
information presented, or notice and report the absence of such evidence. In the ’Citation’, students were 
to look for references to published academic studies, and in ’Source’, students were to report the author or 
group responsible for putting the information on the WWW. Instead of recognizing these separate 
categories, students used ’Evidence’ as a catch-all for any evaluations they made of the resource’s 
credibility. In their published reviews, every student collapsed the three categories of ’Evidence’, 
’Citation’, and ’Source’ into a single category, and reported their evaluation of credibility using the 
’Evidence’ field. This, despite the fact that we had gone to some pains to make the distinctions clear, 
including modeling of the separate kinds of analysis, and inclusion of an on-line glossary linked to the 
WWW form. Helping students recognize different methods of establishing credibility will be a key 
challenge for future WWW reviewing projects. 

Ninety percent (46 of 51) of student reviews contained something under the evidence category, 
coded those 46 responses to see what students were writing in response to the call for evidence 
evidence fields contained more than one of the categories below. 

26% (12 of 46) gave information about scientific studies supporting claims in the WWW resource. 

41% (19 of 46) reported that there was no evidence (i.e. description of methods available). 



We 
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15% (7 of 46) provided elaboration of the claims. 

35 % (16 of 46) referred to the source of the information. 

Evidence in WWW pages: While only 26% of student-published reviews described scientific evidence, 
another 41% did report on the absence of evidence. We knew beforehand that many WWW pages did not 
contain enough information about evidence (the practice review site was one such) and felt that helping 
students notice this absence was important. As preparation for their own air pollution studies, we 
wanted students to be consious of the role of methods in supporting claims made about air pollution. 

Elaboration versus evidence: Another difficulty for students was in separating what we would consider 
'evidence' from mere elaboration of claims. A number of students (15%) made the error of providing 
content elaboration in the place of scientific evidence. For example, reviewing aWWW page on carbon 
monoxide poisoning, one pair of students reported as evidence the typical concentration levels of CO 2 
found near gas stoves. Although this is relevant to the claim, and this information does tangentially add 
to the credibility of the claim, this information is not 'evidence' the way we have defined it. This mistake 
is not as foolish as it may seem, because in non-scientific settings, elaboration often does suffice to 
establish credibility. In a composition class, for instance, students are frequently instructed to 'support 
your points' through elaboration, rather than scientific methodology. It is the somewhat unique 
epistemology of science that requires description of methods. Teaching high school students to look for 
and critique scientific evidence may be a worthy purpose for student-published WWW reviews. 

Source versus evidence: The most frequent confusion about the evidence category, as mentioned before, 
was students putting information in the evidence field that we had wanted them to put in the 'source' or 
'citation' fields. In 35% of the reviews students used the 'Evidence' field to report who conducted the 
studies rather than describing the scientific studies themselves. Again, this confusion is understandable 
when we consider that this is the normal way that high school students (and almost everyone else, unless 
expert in a field) have for establishing scientific credibility of source material. 

Our favorite example of reporting source information instead of evidence, (from a student who 
apparently did understand the difference) was this one: "They don't have any evidence for that claim, 
they just have the facts written down. They're the EPA, they can do that." This is a somewhat circular 
claim ('because the EPA presents them, they are facts') that nevertheless has some truth to it. Even if the 
EPA was somehow fabricating data or using unsound methods, it is highly unlikely that a high school 
student, or any other non-expert would be able to detect this. (This same student wrote, in their 
description field, "If hopefully they are not telling any lies then this is an excellent resource.") In the 
future we may emphasize identifying the source of information as the primary means of establishing 
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credibility, although we will continue to ask students to at least notice when methods are or are not 
described. 

Overall, in examining the evidence’ section of the published WWW reviews we are encouraged that 
students did conduct some critical evaluation of WWW pages, a skill which, as noted before, is not 
frequently demanded of high school students in science. In the future, the main challenge will be to 
separate distinct ways of establishing credibility. 

'Source' field 

Although the separate Source ’field’ was universally ignored by the students this project, it is still 
valuable to examine who the publishing sources of student documents were, and discuss some of the 
emergent issues of evaluating sources. 

Students wrote reviews of WWW pages from a nicely balanced collection of sources, including 
government, commercial, and nonprofit-sponsored pages (figure 4). The diversity of sources shows the 
richness of the WWW, and also highlights the need for critical evaluation skills. 



Government 

(USA) 


Government 

(foreign) 


University 


Nonprofit 

organization 


Commercial 


Commercial 
information 
provider 
(magazine or 
reference) 


18 


6 


8 


7 


9 


3 



figure 4 Distribution of sources reviewed by students. 



A few issues related to this variety of sources seem worth discussing in some detail. 



Ambiguity of sources 

it is often very difficult to correctly attribute the source of information found on the WWW. Often, 
scientific information is given with no attributed source. Sometimes, even the publisher of a WWW page 
is difficult to determine. Students also may be confused when there are multiple sources that they need 
to report. For instance, the sample review page used in this project gives information from a study by the 
American Cancer Society and Harvard University, published on the WWW by a private nonprofit 
environmental advocacy group, giving three potential ’sources’ for this information. 
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A fairly simple skill, that of truncating the URL, should probably be taught to students using the WWW. 
Often, students will follow a link from a page of search hits, and are deposited into the middle of a WWW 
site, with no easily-perceived source. The source can often be found by following truncating the URL. 
This technique is helpful for findingthe WWW publisher of information, although the problem of 
attributing scientific data often remains. 



The dns- 

travel to this address (http://www.nrdc.org) to 
find out who runs the server where this site is 



published. The .org ending is a clue that this is a 
non-government, nonprofit source. 




http:// www.nrdc.org /nrdcpro/bt/tableGU.html 



t \ 

Standard WWW File path names. May be truncated to 

command reveal higher-level pages 

(e.g. http://www.nrdc.org/nrdcpro). 



figure 5 Truncating URLs can be a key methods of finding the source of a WWW page 



Bias of sources 



How should high school students judge the potential biases of the information they find? This is perhaps 
most easily done when the source is commercial. One good site reviewed by students in this pilot project 
was a fact scheet about carbon monoxide published by the American Automobile Manufacturers 
Association^ (AAMA). The information was helpful, but it was also good for students to realize that the 
publisher of this source has a vested financial interest in environmental regulation. 

It is more difficult to judge the potential biases of a nonprofit agency. It is helpful for students to spend 
time in an agency’s WWW site trying to get an overall sense of what the agency's mission and goals are. 

If possible, it is also good for students to try to pin down a nonprofit’s source of funding (although this is 
often not possible on the WWW.) 

Most difficult, perhaps, is describing the potential biases of a government source. Should we tell students 
that government agencies are 'bias-free? If not, what are it’s biases? Most citizens would recognize that a 



2 http://www.aama.com/briefs/autoemissions3.html 
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government agency does have biases towards perpetuating its own bureaucracy and keeping its mission 
in the public eye. However, it is not clear how this subtle bias might relate to the routine data collection 
and analysis done by government agencies. 

In general, in this pilot study project, our goals was to simply have students notice that information on 
the WWW was published by sources with potential biases. We did not spend much time examining how 
these potential biases might affect the information presented. 

In other situations where more time is available, WWW reviews may be an excellent vehicle with which 
to study how biases affect scientific information. Students may come into such projects with the idea that 
bias is important only if a source has fabricated data outright— otherwise, scientific information is just 
’facts’. We suspect that outright fabrication by legitimate organizations is somewhat rare, however. 

Most bias show up in different ways, which students may learn to recognize. Students can learn to think 
in sophisticated ways about bias by discussing how information may be politicized, and how it may be 
selectively presented. Some nonprofit organizations seem especially prone to politicization, or haphazard 
mixing of political and scientific claims in the same article or page. Students can learn to recognize this 
and should probably learn to treat these sources skeptically. The other biasing tendency is selection of 
information. An agency or company will naturally tend to present information that supports its position. 
The AAMA fact sheet on automobile emissions mentioned earlier is a good example of this. The 
information presented, showing that most air pollution comes from cars built before current regulatory 
standards, is factual and helpful, but also tends to support a political position beneficial to the AAMA 
(that further regulation won’t reduce air pollution much until older cars are scrapped). An information- 
savvy student should not necessarily assume that the AAMA ’made up’ this information (we expect that 
they didn't) but should suspect that there may be other information to support other opinions on this 
matter, and should use other sources to find those opinions. 

Organization and appearance 

The next two categories of review comments are organization and appearance. As with previous 
categories, we wondered beforehand whether students would be able to make distinctions between these 
two similar concepts. In this case, our concerns were unfounded, as students were quite willing and able 
to evaluate organization and appearance as separate concepts. However, they did not seem to have a lot 
to say about either of them. 82% (42 of 51) of reviews containted an entry for appearance, and 82% (42 of 
51) containted an entry for organization. The typical entry was quite short, averaging 1.4 lines of text for 
’appearance’ and 1.6 lines oftext for ’organization’ fields. 

Appearance. We expected appearance would be one of the easiest review categories, because students 
generally already have some opinion about the aesthetic value of the WWW pages they are looking at. 
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We also expected that students' evaluation of appearance would depend largely on the amount and 
quality of graphics, (as opposed to subtler features of layout) and we were correct on all counts. Overall, 
students were unimpressed with the graphical quality of the pages they viewed, but typically viewed this 
as an acceptable feature of scientific articles. The comment from one student pair "it isn’t amazingly 
gorgeous, but it gets the job done" echoed the sentiment of many students. 

One way to increase the sophistication of students' graphics evaulations may be to separate graphics 
more clearly by function. Some graphics serve a purely decorative function, some serve to reinforce or 
illustrate the text content, and some (such as graphs) present original content. In the future we hope to 
help students become more sophisticated in identifying these characteristics. 

Organization. This category asked students to focus on how the information on a WWW page was 
arranged and made acessible. 'Organization' is a much less common category of critique for a high school 
student to make. However, the group of students participating in this pilot did have some experience 
because the topic was covered in class a year before, in a project where students created WWW pages. 
Perhaps due partly to this previous experience, students were able to make some good critiques of 
organization, commenting on the use of headers, length, hyperlinks, and other relevant features. 

Description 

The description field represents the most interesting writing task in the WWW reviews. Having analyzed 
a source according to a series of distinct categories, students were asked to then synthesize these 
comments into a description of the resource which would appear by itself on the class’ summary page, 
(see figure 1, previously displayed). 

The description is thus part of the review, but also could stand alone as a 'mini' review, similar to an 
abstract. A user reading the description can select one link and go directly to the reviewed resource, or 
select a different link to read the full reviews (figure 2). 

Although the description is the first part of a student review that digital library patrons would read, we 
arranged the html form so that students composed the description last. By the time that the students 
wrote the description they would hopefully have thought critically about the other aspects of the 
resource, and could synthesize these separate analyses into a few well-chosen sentences. The features 
that we hoped to see in the descriptions were syntheses of two or more evalution categories (e.g. both 
content and credibility), 'meta' level evlauation of the source, and a writing style that indicated they were 
thinking about potential audiences for their review. 
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Student review descriptions did contain some synthesis and evaluation. Thirty percent (13/43) of the 
review descriptions synthesized information from two or more review categories. The descriptions that 
did not synthesize content tended to be either content summaries or simple evaluative statements. Fifty- 
eight percent ( 25/43) of reviews containing descriptions included at least one evaluative statement. 
Evaluative statements were often the first line of the reviews ("this was a very good resource"), or close 
paraphrases of their own review comments ("This page had attractive graphics"). 

The most difficult evaluation/ synthesis task seemed to involve evaluating content. Even the best of the 
students’ descriptions tend to contain either content summaries or evaluative statements, but rarely do a 
good job of both summarizing and evaluating content. In the future, helping students write such 
syntheses will be a pedagogical goal. 

A high-quality description does more than simply synthesize and evaluate. The review writer should 
select information very deliberately, based on the needs of an intended audience. We are still developing 
methods, including interview and ’think-aloud’ studies, for determining to what extent students consider 
the needs of their potential WWW audience and how these considerations play out in their written 
products. 




figure 6 Conceptual model showing how comments from six categories are synthesized, with an 
audience in mind, into a single description 

Student opinions of WWW reviews 

Survey results on WWW reviews- some students reported that writing reviews madethem more 
conscious of use of evidence in their lab reports. 

Students’ opinions on the WWW reviews activity was split. We asked students six likert survey items 
(Appendix A) about whether they thought writing WWW reviews was valuable, and whether they 
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thought the learned from it. We found something of a bimodal split (figure 7), with some students quite 
positive about all aspects of the review-writing, and some quite negative. We have not yet found any 
variable that seems to account for this split. 



Student opinions of WWW 
reviews activity 




'i-'i-'i-CVJOsJOsJOsJCOCOCOCO^t^t'^- 



Average likert survey responses 



figure 7 Bimodal distribution of student responses, averaged across six likert items, to questions 
about the worth of writing WWW reviews. Higher responses are more positive. 

Examining the average responses, we found slighty positive responses on two items. Students thought 
that writing WWW reviews made them more conscious of their own use of evidence and citation, (3.1 on 
5 pt scale) and said that in designing their own WWW pages they tried to avoid some of the mistakes or 
bad features of resources they had reviewed (3.2 on 5 pt scale). 

Discussion 

This project was a pilot study of students-published resources reviews. These reviews were an exercise in 
critical evaluation, as well as being an authentic Value-added' contribution to the WWW. We provided 
an online html form to allow publishing of these reviews, as well as instructional support in the form of 
modeling and a practice review. 

Reviews differed from summaries that were composed by the same class in thes same time frame. The 
'summaries' contained more content summary, while the ’reviews' contained more ’meta' level statements. 

A key difficulty for students was in separating three closely-related categories of 'Evidence', 'Citation', 
and 'Source*. Students reported all of these in the 'Evidence* field, along with some 'evidence' which was 
actually mere elaboration, and ignored the other two fields. 
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Future plans 



At this writing, the WWW reviews have already been used a second time by the same cohort of students, 
[http://chs-web.neb.net/diseases/reviews/] although these reviews have not yet been analyzed. For 
this second project, we collapsed the three fields related to credibility into one field, 'Source'. Although it 
would be worthwhile to teach students to separate different ways of establishing credibility, that goal did 
not fit the overall goals of this project. We did take on the challenge of helping students evaluate 
'Appearance' and 'Organization' in more depth, by giving them direct instruction about different design 



We are also in the process of developing software, called ShaReview, which will take WWW reviews to 
the next level of technological sophistication. ShaReview writes student-composed reviews to a 



capability, we hope will allow students both more control and more ownership over what they have 
written. ShaReview is also being designed so that it can be easily modified by teachers for disparate 
purposes, and we hope to distribute it free to K-12 teachers. 



principles. 



FileMaker Pro database, where the can be edited by students, and searched by outside users. ShaReview 
also links each student- written review to a student 'home page’. This feature, along with the editing 
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Appendix A 



Six likert survey questions about WWW reviews, asked of students after the air pollution laboratory 
reports project 

WWW reviews 

A. Do you think that writing and posting WWW reviews online was a worthwhile part of this project? 

5 (definitely yes) 4 3 2 1 (definitely no) 

B. Did reviewing other WWW pages help you become more analytical in your reading of scientific 
sources? 

5 (definitely yes) 4 3 2 1 (definitely no) 

C. Did any of the WWW pages you reviewed influence the way you designed your own WWW page? 

5 (definitely yes) 4 3 2 1 (definitely no) 

D. After reviewing other pages on the WWW, were you more conscious of your own use of evidence and 
citation (in your project reports)? 

5 (definitely yes) 4 3 2 1 (definitely no) 

E. After reviewing other pages on the WWW, were you more conscious of how other people might read 
the report your group was publishing? 

5 (definitely yes) 4 3 2 1 (definitely no) 

F. In designing your own WWW page, did you try to avoid some of the mistakes or bad features you saw 
on other WWW pages that you reviewed? 

5 (definitely yes) 4 3 2 1 (definitely no) 
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